Partitioned Linear Programming Approximations for MDPs
نویسندگان
چکیده
Approximate linear programming (ALP) is an efficient approach to solving large factored Markov decision processes (MDPs). The main idea of the method is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming (LP). This paper proposes a new ALP approximation. Comparing to the standard ALP formulation, we decompose the constraint space into a set of low-dimensional spaces. This structure allows for solving the new LP efficiently. In particular, the constraints of the LP can be satisfied in a compact form without an exponential dependence on the treewidth of ALP constraints. We study both practical and theoretical aspects of the proposed approach. Moreover, we demonstrate its scale-up potential on an MDP with more than 2 states.
منابع مشابه
Overview of Linear Program Approximations for Factored Continuous and Hybrid-State Markov Decision Processes
Approximate linear programming (ALP) is as one of the most promising methods for solving complex factored MDPs. The method was applied first to tackle problems with discrete state variables. More recently the ALP methods that can solve MDPs with continuous and hybrid (both continuous and discrete) variables have emerged. This paper briefly reviews the work on ALP methods for such problems.
متن کاملLinear Program Approximations for Factored Continuous-State Markov Decision Processes
Approximate linear programming (ALP) has emerged recently as one of the most promising methods for solving complex factored MDPs with finite state spaces. In this work we show that ALP solutions are not limited only to MDPs with finite state spaces, but that they can also be applied successfully to factored continuous-state MDPs (CMDPs). We show how one can build an ALP-based approximation for ...
متن کاملSymbolic Dynamic Programming for Discrete and Continuous State MDPs
Many real-world decision-theoretic planning problems can be naturally modeled with discrete and continuous state Markov decision processes (DC-MDPs). While previous work has addressed automated decision-theoretic planning for DCMDPs, optimal solutions have only been defined so far for limited settings, e.g., DC-MDPs having hyper-rectangular piecewise linear value functions. In this work, we ext...
متن کاملOn the Smoothness of Linear Value Function Approximations
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. It is known that the solution to this convex optimization problem minimizes the...
متن کاملLinearly-solvable Markov decision problems
Advances in Neural Information Processing Systems 2006 We introduce a class of MPDs which greatly simplify Reinforcement Learning. They have discrete state spaces and continuous control spaces. The controls have the effect of rescaling the transition probabilities of an underlying Markov chain. A control cost penalizing KL divergence between controlled and uncontrolled transition probabilities ...
متن کامل